📈 The Trajectory of LLM Capabilities

📈 A Seven-Year Sweep

The pace of capability change over the last seven years is worth pausing on. Below is a deliberately compressed history. Each row marks a moment when the answer to “what can these systems do?” shifted enough that the previous answer became wrong.

🔬 The May 2026 Frontier — Closed-Weights Models

As of mid-May 2026, the closed-weights frontier comprises six families of models, all from large well-resourced labs. Each has slightly different strengths, and the leaderboards shuffle every few weeks. The numbers below are accurate at time of writing and will be wrong within months — that itself is a teaching point.

🔒 The May 2026 Frontier — Open-Weights Models

A meaningful part of the 2026 story is that open-weights models have closed most of the gap. This wasn't true in 2023–24, where closed labs had a clear lead on every important capability. As of May 2026, an open-weights model leads the world on competitive programming, and matches the closed frontier on most agentic and coding benchmarks at substantially lower cost.

🧮 What These Benchmarks Actually Measure

Benchmark scores are now the standard currency of AI capability claims. They are useful, but only if you know what each one tests. A 90% on AIME and a 50% on FrontierMath sound very different but might describe the same model on tasks of very different difficulty.

🚫 What Benchmarks Don't Tell You

Benchmark scores are easy to read and easy to compare. They give the illusion of an objective measure of capability. They are, in fact, a partial and contested measure — and three specific failures of benchmarks are worth knowing before you read any model announcement uncritically.

1. Goodhart's Law: Benchmarks Are Not Real-World Use

Charles Goodhart, a British economist, originally articulated the principle in 1975 in the context of monetary policy: any observed statistical regularity tends to collapse once pressure is placed upon it for control purposes. The popular formulation above is from anthropologist Marilyn Strathern in 1997.

The application to AI benchmarks is direct. The benchmarks were originally proxies for capability — if a model scores well on FrontierMath, that's evidence the model can do research-level mathematics. But once benchmarks become the public yardstick for model quality, labs train and tune specifically to perform well on them. The benchmark stops being a proxy for capability and becomes a target the model is optimised to hit. The resulting score is no longer a good measure of the underlying capability the benchmark was meant to track.

A concrete consequence: a model that scores 87.6% on SWE-bench Verified is genuinely good at GitHub-issue-style coding tasks. It may or may not be good at the coding work you need to do. A model that scores 52.4% on FrontierMath is genuinely capable on Fields-medal-curated mathematics problems. It may or may not be useful for the specific mathematical questions in your field. The benchmark measures performance on the benchmark; that's all it can measure.

This is why Sub-Lesson 9.6 asks you to test current frontier models on tasks from your own field, rather than rely on published benchmark scores. The benchmarks tell you what the labs have optimised for. They tell you very little about whether the model will work for what you need.

2. Benchmark Contamination — Training-Data Leakage

A benchmark only measures capability if the model hasn't already seen the answers. The internet contains most published benchmarks. Models trained on web-scale data have, in many cases, seen the benchmark questions and answers during training — producing inflated scores that don't reflect generalised capability.

The implication: a model's reported MMLU score may overstate its general reasoning ability, possibly substantially. Comparable concerns apply to AIME, GSM8K, HumanEval, and most popular benchmarks — all of which have appeared in training corpora.

Three responses have emerged:

• Held-out / private benchmarks. FrontierMath was deliberately constructed as 350 entirely new and unpublished problems. The Tier 4 set has 20 problems explicitly held out from OpenAI (the funder) so a private set remains for honest evaluation; the leaderboard scores reported in this lesson are evaluated on the private sets. CharXiv (Week 8) used a similar design.
• Cleaned and refreshed benchmarks. MMLU-Pro (Wang et al. 2024) increased question difficulty and expanded answer choices to make memorisation less helpful. MMLU-Redux (2025) cleaned ambiguous questions and incorrect answers from the original.
• Dynamic benchmarks. Benchmarks regenerated continuously based on training cutoffs, so contamination is structurally impossible. LatestEval (AAAI 2024) is one example.

When you read a benchmark score in a model announcement, ask: was this evaluation done on a private/held-out set, or on a public benchmark that was probably in the training data?

3. What Benchmarks Don't Test — Low-Resource Languages and African Contexts

Almost every major benchmark in the table above — FrontierMath, GPQA Diamond, SWE-bench Verified, AIME, MCP-Atlas, HLE — is administered in English. A model that excels on these benchmarks is genuinely good at English-language reasoning on topics well-represented in English-language training data. The leaderboard scores tell you very little about how the model performs on questions in Setswana, on legal reasoning grounded in South African law, on medical questions framed for a Nigerian or Kenyan clinical context, or on mathematics problems written by African educators for African students.

For UCT students, this matters directly: the benchmark scores you see in news coverage may not predict how the same model performs on your research questions, your students' questions, or your local context.

Africa-specific benchmarks have begun to emerge in response to the gap. They are useful but not yet widely cited in model announcements:

• AfroBench (ACL 2025) — comprehensive evaluation of LLMs on African languages across multiple NLP tasks.
• IrokoBench — benchmark designed specifically for African languages in the LLM era.
• AfriSpeech-Dialog (NAACL 2025) and Intron Sahara v2 (2026, 57 languages) — African-language ASR benchmarks (covered in Week 8).
• Nahabwe et al. (2025), Benchmarking Automatic Speech Recognition Models for African Languages, arXiv:2512.10968 — covered in Week 8.

The pattern: for well-represented African languages (Swahili, Hausa, Yoruba, Afrikaans), frontier models perform reasonably. For under-represented languages (Setswana, Sepedi, isiXhosa, isiZulu in many tasks), performance falls off substantially. Performance differences between languages are larger than performance differences between frontier models.

📚 Centrepiece Reading: Gowers on ChatGPT 5.5 Pro

Tim Gowers is a Fields medallist and a long-time chronicler of mathematical practice. In May 2026 he posted an account of an experiment in which he used ChatGPT 5.5 Pro to attack open problems from a paper by Mel Nathanson on additive number theory.

🎯 The Move This Sub-Lesson Makes